{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 深度学习部分 第一周作业"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "作业内容：\n",
    "使用 tensorflow，构造并训练一个神经网络，在测试机上达到超过 98%的准确率。\n",
    "在完成过程中，需要综合运用目前学到的基础知识：\n",
    " 深度神经网络\n",
    " 激活函数\n",
    " 正则化\n",
    " 初始化\n",
    "并探索如下超参数设置：\n",
    " 隐层数量\n",
    " 各隐层中神经元数量\n",
    " 学习率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载MNIST数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data\\train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data\\train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data\\t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 启动TensorFlow InteractiveSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 建立一个Softmax回归模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 占位符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, shape=[None, 784])\n",
    "y_ = tf.placeholder(tf.float32, shape=[None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 变量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.zeros([784,10]))\n",
    "b = tf.Variable(tf.zeros([10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#变量的赋值\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 预测类和损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#实现回归模型\n",
    "y = tf.matmul(x,W) + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#损失函数是目标和应用于模型预测的softmax激活函数之间的交叉熵\n",
    "cross_entropy = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#运行train_step时返回的参数会用于更新梯度下降。 因此，训练模型可以通过重复运行train_step来完成。\n",
    "for _ in range(1000):\n",
    "  batch = mnist.train.next_batch(100)\n",
    "  train_step.run(feed_dict={x: batch[0], y_: batch[1]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9178\n"
     ]
    }
   ],
   "source": [
    "#预测\n",
    "correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print(accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 隐藏层探索"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "添加了两个隐藏层，分别有500和300个神经元，这样包括输入输出层，总共4层神经网络\n",
    "其中：\n",
    "（1）隐藏层初始化函数建议使用tf.truncated_normal()（截短的随机数）类型，而非前文中的tf.zero()（初始化为零）类型\n",
    "（2）中间层的激活函数，本文使用tanh（双曲正切函数），建议读者可以尝试运用ReLU函数或者Sigmoid函数，比较一下输出结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W1 = tf.Variable(tf.truncated_normal([784, 500], stddev=0.1), name='W1')\n",
    "b1 = tf.Variable(tf.zeros([500]) + 0.1, name='b1')\n",
    "L1 = tf.nn.tanh(tf.matmul(x, W1) + b1, name='L1')\n",
    "\n",
    "W2 = tf.Variable(tf.truncated_normal([500, 300], stddev=0.1), name='W2')\n",
    "b2 = tf.Variable(tf.zeros([300]) + 0.1, name='b2')\n",
    "L2 = tf.nn.tanh(tf.matmul(L1, W2) + b2, name='L2')\n",
    "\n",
    "W3 = tf.Variable(tf.truncated_normal([300, 10], stddev=0.1), name='W3')\n",
    "b3 = tf.Variable(tf.zeros([10]) + 0.1, name='b3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 计算预测结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prediction = tf.nn.softmax(tf.matmul(L2, W3) + b3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 计算损失值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean(tf.square(y - prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 初始化optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.2 #学习率\n",
    "optimizer =  tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "\n",
    "# 结果存放在一个布尔型列表中\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(prediction, 1))  \n",
    "\n",
    "# 求准确率\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 指定迭代次数，并在session执行graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter0, Testing accuracy:0.8898\n",
      "Iter2, Testing accuracy:0.922\n",
      "Iter4, Testing accuracy:0.9307\n",
      "Iter6, Testing accuracy:0.9364\n",
      "Iter8, Testing accuracy:0.9412\n",
      "Iter10, Testing accuracy:0.945\n",
      "Iter12, Testing accuracy:0.9495\n",
      "Iter14, Testing accuracy:0.9513\n",
      "Iter16, Testing accuracy:0.9534\n",
      "Iter18, Testing accuracy:0.9559\n",
      "Iter20, Testing accuracy:0.9567\n"
     ]
    }
   ],
   "source": [
    "# 每个批次送100张图片\n",
    "batch_size = 100\n",
    "# 计算一共有多少个批次\n",
    "n_batch = mnist.train.num_examples // batch_size\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch in range(21):\n",
    "        for batch in range(n_batch):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "            sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})\n",
    "\n",
    "        test_acc = sess.run(accuracy, feed_dict={x: mnist.test.images, y: mnist.test.labels})\n",
    "\n",
    "        if epoch % 2 == 0:\n",
    "            print(\"Iter\" + str(epoch) + \", Testing accuracy:\" + str(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "由此可见添加隐藏层，可以提高准确率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学习率探索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter0, Testing accuracy:0.91\n",
      "Iter2, Testing accuracy:0.9343\n",
      "Iter4, Testing accuracy:0.9436\n",
      "Iter6, Testing accuracy:0.9501\n",
      "Iter8, Testing accuracy:0.9534\n",
      "Iter10, Testing accuracy:0.9581\n",
      "Iter12, Testing accuracy:0.9599\n",
      "Iter14, Testing accuracy:0.9614\n",
      "Iter16, Testing accuracy:0.9628\n",
      "Iter18, Testing accuracy:0.9655\n",
      "Iter20, Testing accuracy:0.9672\n"
     ]
    }
   ],
   "source": [
    "learning_rate = 0.4 #学习率\n",
    "optimizer =  tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "# 结果存放在一个布尔型列表中\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(prediction, 1))  \n",
    "# 求准确率\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "# 每个批次送100张图片\n",
    "batch_size = 100\n",
    "# 计算一共有多少个批次\n",
    "n_batch = mnist.train.num_examples // batch_size\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch in range(21):\n",
    "        for batch in range(n_batch):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "            sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})\n",
    "\n",
    "        test_acc = sess.run(accuracy, feed_dict={x: mnist.test.images, y: mnist.test.labels})\n",
    "\n",
    "        if epoch % 2 == 0:\n",
    "            print(\"Iter\" + str(epoch) + \", Testing accuracy:\" + str(test_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter0, Testing accuracy:0.9193\n",
      "Iter2, Testing accuracy:0.9415\n",
      "Iter4, Testing accuracy:0.9513\n",
      "Iter6, Testing accuracy:0.9556\n",
      "Iter8, Testing accuracy:0.9599\n",
      "Iter10, Testing accuracy:0.9646\n",
      "Iter12, Testing accuracy:0.966\n",
      "Iter14, Testing accuracy:0.9669\n",
      "Iter16, Testing accuracy:0.9698\n",
      "Iter18, Testing accuracy:0.9701\n",
      "Iter20, Testing accuracy:0.9706\n"
     ]
    }
   ],
   "source": [
    "learning_rate = 0.6 #学习率\n",
    "optimizer =  tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "# 结果存放在一个布尔型列表中\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(prediction, 1))  \n",
    "# 求准确率\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "# 每个批次送100张图片\n",
    "batch_size = 100\n",
    "# 计算一共有多少个批次\n",
    "n_batch = mnist.train.num_examples // batch_size\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch in range(21):\n",
    "        for batch in range(n_batch):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "            sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})\n",
    "\n",
    "        test_acc = sess.run(accuracy, feed_dict={x: mnist.test.images, y: mnist.test.labels})\n",
    "\n",
    "        if epoch % 2 == 0:\n",
    "            print(\"Iter\" + str(epoch) + \", Testing accuracy:\" + str(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由此可见增加学习率，可以提高准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter0, Testing accuracy:0.9325\n",
      "Iter2, Testing accuracy:0.9563\n",
      "Iter4, Testing accuracy:0.9631\n",
      "Iter6, Testing accuracy:0.9686\n",
      "Iter8, Testing accuracy:0.9712\n",
      "Iter10, Testing accuracy:0.973\n",
      "Iter12, Testing accuracy:0.9739\n",
      "Iter14, Testing accuracy:0.9742\n",
      "Iter16, Testing accuracy:0.9747\n",
      "Iter18, Testing accuracy:0.9755\n",
      "Iter20, Testing accuracy:0.9758\n",
      "Iter22, Testing accuracy:0.9759\n",
      "Iter24, Testing accuracy:0.9767\n",
      "Iter26, Testing accuracy:0.9766\n",
      "Iter28, Testing accuracy:0.9769\n",
      "Iter30, Testing accuracy:0.9769\n",
      "Iter32, Testing accuracy:0.9761\n",
      "Iter34, Testing accuracy:0.9773\n",
      "Iter36, Testing accuracy:0.9772\n",
      "Iter38, Testing accuracy:0.9775\n",
      "Iter40, Testing accuracy:0.9773\n",
      "Iter42, Testing accuracy:0.9778\n",
      "Iter44, Testing accuracy:0.9782\n",
      "Iter46, Testing accuracy:0.9768\n",
      "Iter48, Testing accuracy:0.9779\n",
      "Iter50, Testing accuracy:0.9774\n"
     ]
    }
   ],
   "source": [
    "learning_rate = 1.5 #学习率\n",
    "optimizer =  tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "# 结果存放在一个布尔型列表中\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(prediction, 1))  \n",
    "# 求准确率\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "# 每个批次送100张图片\n",
    "batch_size = 100\n",
    "# 计算一共有多少个批次\n",
    "n_batch = mnist.train.num_examples // batch_size\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch in range(51):\n",
    "        for batch in range(n_batch):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "            sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})\n",
    "\n",
    "        test_acc = sess.run(accuracy, feed_dict={x: mnist.test.images, y: mnist.test.labels})\n",
    "\n",
    "        if epoch % 2 == 0:\n",
    "            print(\"Iter\" + str(epoch) + \", Testing accuracy:\" + str(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "增加迭代次数，可以提高准确率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建一个多层卷积网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 权重初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def weight_variable(shape):\n",
    "  initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "  return tf.Variable(initial)\n",
    "\n",
    "def bias_variable(shape):\n",
    "  initial = tf.constant(0.1, shape=shape)\n",
    "  return tf.Variable(initial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 卷积和池化"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "conv2d(\n",
    "    input,\n",
    "    filter,\n",
    "    strides,\n",
    "    padding,\n",
    "    use_cudnn_on_gpu=True,\n",
    "    data_format='NHWC',\n",
    "    name=None\n",
    ")\n",
    "1、input：张量。必须是以下类型之一：half，float32。一个四维张量。维度顺序根据data_format的值来解释，详见下文。\n",
    "2、filter：张量。必须具有与输入相同的类型。形状的四维张量[filter_height，filter_width，in_channels，out_channels]\n",
    "3、strides：整数列表。一维长度的张量4.输入每个维度的滑动窗口的步幅。维度顺序由data_format的值决定，详见下文。\n",
    "4、padding：来自“SAME”，“VALID”的字符串。要使用的填充算法的类型。\n",
    "5、use_cudnn_on_gpu：一个可选的布尔。默认为True。\n",
    "6、data_format：来自“NHWC”，“NCHW”的可选字符串。默认为“NHWC”。指定输入和输出数据的数据格式。使用默认格式“NHWC”，数据按照[batch，height，width，channels]的顺序存储。或者，格式可以是“NCHW”，数据存储顺序为：[batch，channels，height，width]。\n",
    "7、name：操作的名称（可选）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#步幅大小为1，，并在周围填充零，以便输出与输入大小相同\n",
    "def conv2d(x, W):\n",
    "  return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "max_pool(\n",
    "    value,\n",
    "    ksize,\n",
    "    strides,\n",
    "    padding,\n",
    "    data_format='NHWC',\n",
    "    name=None\n",
    ")\n",
    "\n",
    "1、value：由data_format指定的格式的4维张量。\n",
    "2、ksize：4元素的1维 int型张量。 输入张量表示窗口每个维度的大小。\n",
    "3、strides：4元素的1维int型张量。 输入张量表示滑动窗口每个维度的步幅。\n",
    "4、padding：一个字符串，可以是’VALID’ 或 ‘SAME’。\n",
    "5、data_format：一个字符串。 支持“NHWC”，“NCHW”和“NCHW_VECT_C”。\n",
    "6、name：操作的可选名称。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#pooling是超过2x2的max pooling\n",
    "def max_pool_2x2(x):\n",
    "  return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],\n",
    "                        strides=[1, 2, 2, 1], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第一卷积层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#它将由卷积组成，然后是max pooling。 卷积将为每个5x5 patch计算32个特征。 它的权重张量是[5,5,1,32]的形状。\n",
    "W_conv1 = weight_variable([5, 5, 1, 32])\n",
    "b_conv1 = bias_variable([32])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#为了应用该层，首先将x重塑为4维张量，第二维和第三维对应于图像的宽度和高度，并且最后一个维度对应于色彩通道的数量。\n",
    "x_image = tf.reshape(x, [-1, 28, 28, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#然后，我们x_image与权重张量进行卷积，加上偏差，应用ReLU函数，最后使用max pooling。 max_pool_2x2方法将图像大小减小到14x14。\n",
    "h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)\n",
    "h_pool1 = max_pool_2x2(h_conv1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第二卷积层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#第二层将为每个5x5 patch有64个特征。\n",
    "W_conv2 = weight_variable([5, 5, 32, 64])\n",
    "b_conv2 = bias_variable([64])\n",
    "\n",
    "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n",
    "h_pool2 = max_pool_2x2(h_conv2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 密集连接层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#现在图像尺寸已经减小到7x7，我们添加了一个1024个神经元的全连接图层，以允许在整个图像上进行处理。 将pooling层中的张量重塑为一批向量，乘以权重矩阵，添加一个偏差，并应用一个ReLU。\n",
    "W_fc1 = weight_variable([7 * 7 * 64, 1024])\n",
    "b_fc1 = bias_variable([1024])\n",
    "\n",
    "h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n",
    "h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropout"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "dropout(\n",
    "    x,\n",
    "    keep_prob,\n",
    "    noise_shape=None,\n",
    "    seed=None,\n",
    "    name=None\n",
    ")\n",
    "\n",
    "1、x：浮点张量。\n",
    "2、keep_prob：与x相同类型的张量。 每个元素被保留的概率。\n",
    "3、noise_shape：int32类型的一维张量，表示随机生成的保留/丢弃标志。\n",
    "4、seed：一个Python整数。 用于创建随机种子。 有关行为，请参阅tf.set_random_seed。\n",
    "5、name：此操作的名称（可选）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#TensorFlow的tf.nn.dropout可以自动处理缩放神经元输出和掩蔽它们，所以dropout只是在没有任何附加缩放的情况下工作\n",
    "keep_prob = tf.placeholder(tf.float32)\n",
    "h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读出层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#添加一个图层，就像上面的一层softmax回归一样\n",
    "W_fc2 = weight_variable([1024, 10])\n",
    "b_fc2 = bias_variable([10])\n",
    "\n",
    "y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练和评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.16\n",
      "step 100, training accuracy 0.84\n",
      "step 200, training accuracy 0.86\n",
      "step 300, training accuracy 0.92\n",
      "step 400, training accuracy 0.92\n",
      "step 500, training accuracy 0.94\n",
      "step 600, training accuracy 0.96\n",
      "step 700, training accuracy 0.96\n",
      "step 800, training accuracy 0.96\n",
      "step 900, training accuracy 1\n",
      "step 1000, training accuracy 0.92\n",
      "step 1100, training accuracy 0.98\n",
      "step 1200, training accuracy 0.98\n",
      "step 1300, training accuracy 0.96\n",
      "step 1400, training accuracy 1\n",
      "step 1500, training accuracy 0.96\n",
      "step 1600, training accuracy 0.94\n",
      "step 1700, training accuracy 0.96\n",
      "step 1800, training accuracy 0.98\n",
      "step 1900, training accuracy 0.96\n",
      "step 2000, training accuracy 1\n",
      "step 2100, training accuracy 1\n",
      "step 2200, training accuracy 1\n",
      "step 2300, training accuracy 1\n",
      "step 2400, training accuracy 0.98\n",
      "step 2500, training accuracy 0.98\n",
      "step 2600, training accuracy 0.96\n",
      "step 2700, training accuracy 0.98\n",
      "step 2800, training accuracy 1\n",
      "step 2900, training accuracy 0.98\n",
      "step 3000, training accuracy 0.98\n",
      "step 3100, training accuracy 1\n",
      "step 3200, training accuracy 0.92\n",
      "step 3300, training accuracy 0.94\n",
      "step 3400, training accuracy 0.98\n",
      "step 3500, training accuracy 0.96\n",
      "step 3600, training accuracy 1\n",
      "step 3700, training accuracy 1\n",
      "step 3800, training accuracy 1\n",
      "step 3900, training accuracy 0.98\n",
      "step 4000, training accuracy 1\n",
      "step 4100, training accuracy 0.96\n",
      "step 4200, training accuracy 0.98\n",
      "step 4300, training accuracy 1\n",
      "step 4400, training accuracy 0.98\n",
      "step 4500, training accuracy 1\n",
      "step 4600, training accuracy 0.96\n",
      "step 4700, training accuracy 1\n",
      "step 4800, training accuracy 0.98\n",
      "step 4900, training accuracy 1\n",
      "step 5000, training accuracy 0.98\n",
      "step 5100, training accuracy 0.98\n",
      "step 5200, training accuracy 0.98\n",
      "step 5300, training accuracy 1\n",
      "step 5400, training accuracy 0.98\n",
      "step 5500, training accuracy 0.98\n",
      "step 5600, training accuracy 0.94\n",
      "step 5700, training accuracy 1\n",
      "step 5800, training accuracy 1\n",
      "step 5900, training accuracy 1\n",
      "step 6000, training accuracy 1\n",
      "step 6100, training accuracy 1\n",
      "step 6200, training accuracy 1\n",
      "step 6300, training accuracy 1\n",
      "step 6400, training accuracy 0.96\n",
      "step 6500, training accuracy 0.98\n",
      "step 6600, training accuracy 1\n",
      "step 6700, training accuracy 1\n",
      "step 6800, training accuracy 0.98\n",
      "step 6900, training accuracy 1\n",
      "step 7000, training accuracy 0.98\n",
      "step 7100, training accuracy 1\n",
      "step 7200, training accuracy 1\n",
      "step 7300, training accuracy 1\n",
      "step 7400, training accuracy 0.98\n",
      "step 7500, training accuracy 0.98\n",
      "step 7600, training accuracy 1\n",
      "step 7700, training accuracy 1\n",
      "step 7800, training accuracy 0.98\n",
      "step 7900, training accuracy 0.98\n",
      "step 8000, training accuracy 1\n",
      "step 8100, training accuracy 0.98\n",
      "step 8200, training accuracy 1\n",
      "step 8300, training accuracy 1\n",
      "step 8400, training accuracy 1\n",
      "step 8500, training accuracy 1\n",
      "step 8600, training accuracy 1\n",
      "step 8700, training accuracy 1\n",
      "step 8800, training accuracy 1\n",
      "step 8900, training accuracy 1\n",
      "step 9000, training accuracy 1\n",
      "step 9100, training accuracy 0.96\n",
      "step 9200, training accuracy 1\n",
      "step 9300, training accuracy 1\n",
      "step 9400, training accuracy 1\n",
      "step 9500, training accuracy 1\n",
      "step 9600, training accuracy 1\n",
      "step 9700, training accuracy 1\n",
      "step 9800, training accuracy 1\n",
      "step 9900, training accuracy 1\n",
      "step 10000, training accuracy 0.98\n",
      "step 10100, training accuracy 1\n",
      "step 10200, training accuracy 1\n",
      "step 10300, training accuracy 1\n",
      "step 10400, training accuracy 1\n",
      "step 10500, training accuracy 1\n",
      "step 10600, training accuracy 1\n",
      "step 10700, training accuracy 1\n",
      "step 10800, training accuracy 1\n",
      "step 10900, training accuracy 0.98\n",
      "step 11000, training accuracy 1\n",
      "step 11100, training accuracy 0.98\n",
      "step 11200, training accuracy 0.98\n",
      "step 11300, training accuracy 1\n",
      "step 11400, training accuracy 1\n",
      "step 11500, training accuracy 1\n",
      "step 11600, training accuracy 1\n",
      "step 11700, training accuracy 1\n",
      "step 11800, training accuracy 1\n",
      "step 11900, training accuracy 0.98\n",
      "step 12000, training accuracy 1\n",
      "step 12100, training accuracy 1\n",
      "step 12200, training accuracy 1\n",
      "step 12300, training accuracy 1\n",
      "step 12400, training accuracy 1\n",
      "step 12500, training accuracy 0.98\n",
      "step 12600, training accuracy 1\n",
      "step 12700, training accuracy 1\n",
      "step 12800, training accuracy 0.98\n",
      "step 12900, training accuracy 1\n",
      "step 13000, training accuracy 0.98\n",
      "step 13100, training accuracy 1\n",
      "step 13200, training accuracy 1\n",
      "step 13300, training accuracy 1\n",
      "step 13400, training accuracy 1\n",
      "step 13500, training accuracy 1\n",
      "step 13600, training accuracy 1\n",
      "step 13700, training accuracy 1\n",
      "step 13800, training accuracy 0.98\n",
      "step 13900, training accuracy 1\n",
      "step 14000, training accuracy 1\n",
      "step 14100, training accuracy 1\n",
      "step 14200, training accuracy 1\n",
      "step 14300, training accuracy 1\n",
      "step 14400, training accuracy 1\n",
      "step 14500, training accuracy 1\n",
      "step 14600, training accuracy 1\n",
      "step 14700, training accuracy 1\n",
      "step 14800, training accuracy 1\n",
      "step 14900, training accuracy 1\n",
      "step 15000, training accuracy 1\n",
      "step 15100, training accuracy 1\n",
      "step 15200, training accuracy 1\n",
      "step 15300, training accuracy 1\n",
      "step 15400, training accuracy 1\n",
      "step 15500, training accuracy 1\n",
      "step 15600, training accuracy 1\n",
      "step 15700, training accuracy 1\n",
      "step 15800, training accuracy 1\n",
      "step 15900, training accuracy 1\n",
      "step 16000, training accuracy 1\n",
      "step 16100, training accuracy 1\n",
      "step 16200, training accuracy 1\n",
      "step 16300, training accuracy 1\n",
      "step 16400, training accuracy 1\n",
      "step 16500, training accuracy 1\n",
      "step 16600, training accuracy 1\n",
      "step 16700, training accuracy 1\n",
      "step 16800, training accuracy 1\n",
      "step 16900, training accuracy 1\n",
      "step 17000, training accuracy 1\n",
      "step 17100, training accuracy 0.98\n",
      "step 17200, training accuracy 1\n",
      "step 17300, training accuracy 1\n",
      "step 17400, training accuracy 1\n",
      "step 17500, training accuracy 1\n",
      "step 17600, training accuracy 1\n",
      "step 17700, training accuracy 1\n",
      "step 17800, training accuracy 1\n",
      "step 17900, training accuracy 1\n",
      "step 18000, training accuracy 0.98\n",
      "step 18100, training accuracy 1\n",
      "step 18200, training accuracy 1\n",
      "step 18300, training accuracy 1\n",
      "step 18400, training accuracy 1\n",
      "step 18500, training accuracy 1\n",
      "step 18600, training accuracy 1\n",
      "step 18700, training accuracy 1\n",
      "step 18800, training accuracy 1\n",
      "step 18900, training accuracy 1\n",
      "step 19000, training accuracy 1\n",
      "step 19100, training accuracy 1\n",
      "step 19200, training accuracy 1\n",
      "step 19300, training accuracy 1\n",
      "step 19400, training accuracy 1\n",
      "step 19500, training accuracy 1\n",
      "step 19600, training accuracy 1\n",
      "step 19700, training accuracy 1\n",
      "step 19800, training accuracy 1\n",
      "step 19900, training accuracy 1\n",
      "test accuracy 0.9916\n"
     ]
    }
   ],
   "source": [
    "cross_entropy = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))\n",
    "train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)\n",
    "correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "\n",
    "with tf.Session() as sess:\n",
    "  sess.run(tf.global_variables_initializer())\n",
    "  for i in range(20000):\n",
    "    batch = mnist.train.next_batch(50)\n",
    "    if i % 100 == 0:\n",
    "      train_accuracy = accuracy.eval(feed_dict={\n",
    "          x: batch[0], y_: batch[1], keep_prob: 1.0})\n",
    "      print('step %d, training accuracy %g' % (i, train_accuracy))\n",
    "    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})\n",
    "\n",
    "  print('test accuracy %g' % accuracy.eval(feed_dict={\n",
    "      x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集精确度为：0.9916"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
